Skip to content

feat(interferometer): pass data=None on the sparse precomputed-data-term path (Discussion #13 lens parity) - #757

Merged
Jammy2211 merged 1 commit into
mainfrom
feature/interferometer-sparse-precomputed-data-term
Sep 30, 2026
Merged

feat(interferometer): pass data=None on the sparse precomputed-data-term path (Discussion #13 lens parity)#757
Jammy2211 merged 1 commit into
mainfrom
feature/interferometer-sparse-precomputed-data-term

Conversation

@Jammy2211

@Jammy2211 Jammy2211 commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

PyAutoLens parity for phase 1 of GitHub Discussion https://github.com/orgs/PyAutoLabs/discussions/13 ("Streaming visibilities for memory efficiency", HRSAstro), mirroring PyAutoLabs/PyAutoGalaxy#637 on top of PyAutoLabs/PyAutoArray#589. Closes #756.

On the sparse interferometer path a lens fit with no non-linear light profile still allocated two N_vis arrays per likelihood evaluation: profile_visibilities (Visibilities.zeros) and profile_subtracted_visibilities (data - zeros), because FitInterferometer.tracer_to_inversion passed the subtracted data unconditionally. This PR:

  • gates tracer_to_inversion on autogalaxy's uses_precomputed_data_term_from(dataset, galaxies, data, noise_map) (sparse operator carries data_term, the fitted data/noise map are the dataset's own, no galaxy has a non-linear light profile) and passes data=None when it holds, so fast_chi_squared reads the cached scalar and neither visibility property is evaluated on the likelihood path; the checks are identity/type checks, so the branch is fixed at trace time and jit-safe;
  • adds FitInterferometer.inversion_with_data: the inversion, or a shallow copy carrying fit.data when the inversion ran with data=None, for output paths (data_subtracted_dict → subplot_of_mapper); the interferometer visualizer uses it;
  • makes profile_visibilities / profile_subtracted_visibilities cached properties (as profile_image already was and as autogalaxy now has them), so "never evaluated" is observable.

Dense-path and light-profile fits are bit-identical to before.

Witness (1e5-vis NUFFT, 616-pixel mask, Isothermal lens + 15x15 rectangular source, InversionInterferometerSparseNumba): figure_of_merit gated vs gate forced off differs by exactly 0.0; tracemalloc peak per figure_of_merit 3.20 MB gated at both 1e5 and 4e5 visibilities (flat in N_vis, below 16·N_vis bytes) vs 7.59 MB / 21.99 MB ungated; 16.2 ms vs 18.0 / 26.6 ms per evaluation.

API Changes

Additive. New public name FitInterferometer.inversion_with_data. Behaviour change: on the sparse path with no non-linear light profile, fit.inversion.dataset.data is now None (use fit.inversion_with_data or fit.data for outputs). profile_visibilities and profile_subtracted_visibilities are now cached per fit instance. Requires PyAutoArray ≥ the #589 merge and PyAutoGalaxy ≥ the #637 merge (both on main, pending release).
See full details below.

Test Plan

  • pytest test_autolens/interferometer — 29 passed (new: gate true + inversion.dataset.data is None; sparse pixelization-only vs dense at rel 1e-8; inversion_with_data shares the reconstruction and carries fit.data; dense fit returns inversion_with_data is inversion; spies on visibilities_from / Visibilities.zeros record no calls and neither profile property is cached after figure_of_merit; light-profile fit gate false and bit-equal; jax.jit sparse fit matches numpy; gate forced off → figure_of_merit bit-equal)
  • pytest test_autolens — 766 passed, 1 xfailed
  • Red check: with the source edits reverted, 5 of the 11 tests in the file fail (all new/extended)
  • Witness script (numbers above)
  • CI green on unittest 3.12 / 3.13 / nojax
Full API Changes (for automation & release notes)

Added

  • autolens.interferometer.fit_interferometer.FitInterferometer.inversion_with_data -> Optional[aa.AbstractInversion] — the inversion carrying the fitted visibilities as its dataset data, for output/plot paths
  • FitInterferometer._uses_precomputed_data_term (private property) — uses_precomputed_data_term_from(dataset, tracer.galaxies, data, noise_map)

Changed Behaviour

  • FitInterferometer.tracer_to_inversion passes data=None to aa.DatasetInterface when _uses_precomputed_data_term holds (sparse operator with data_term, dataset's own data/noise map, no non-linear light profile); profile_visibilities / profile_subtracted_visibilities are then never evaluated on the likelihood path
  • FitInterferometer.profile_visibilities and FitInterferometer.profile_subtracted_visibilities are cached_property (were plain properties); values unchanged
  • autolens/interferometer/model/visualizer.py passes fit.inversion_with_data to the inversion plotter

Migration

  • Code reading fit.inversion.dataset.data / fit.inversion.data_subtracted_dict on a sparse pixelization-only lens fit should use fit.inversion_with_data (or fit.data).

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code

Jammy2211 @claude Jammy2211
feat(interferometer): pass data=None on the sparse precomputed-data-t…
3ea8fd5 
…erm path (#756)

Mirror PyAutoGalaxy#637 in autolens: gate FitInterferometer.tracer_to_inversion
on uses_precomputed_data_term_from so sparse pixelization-only lens fits build
their inversion with data=None and never allocate profile_visibilities /
profile_subtracted_visibilities per likelihood call. Add inversion_with_data
for output paths, use it in the interferometer visualizer, and cache the two
profile visibility properties. Tests mirror autogalaxy's phase-1 additions.

Phase 1 of https://github.com/orgs/PyAutoLabs/discussions/13.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Jammy2211 Jammy2211 added the pending-release PR queued for the next release build label Sep 30, 2026
@Jammy2211 Jammy2211 mentioned this pull request Sep 30, 2026
Closed
@Jammy2211

Jammy2211 commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

Independent review (Codex gpt-6-astra) — this PR + phase 1 (PyAutoArray#589, PyAutoGalaxy#637)

Three read-only Codex reviews (one per repo) returned 6 findings = 4 distinct claims. Every claim was reproduced with scripts before being reported here; nothing has been edited in response yet.

# Claim Verdict Where Real exposure
B fit.inversion.data_subtracted_dict breaks when the inversion ran with data=None (gate on): multi-object → TypeError: unsupported operand type(s) for -: 'NoneType' and 'complex' at autoarray/inversion/inversion/abstract.py:885; single mapper → {mapper: None} and subplot_of_mapper raises ValueError at plot/array.py:206. The inversion_plots.py:87/:397 handlers catch only (AttributeError, KeyError). Reproduced — introduced by phase 1 / this PR autoarray + plotters Yes. autogalaxy_workspace scripts/interferometer/features/pixelization/fit.py:321-322 passes fit.inversion on a sparse dataset (live on PyAutoGalaxy main now); autolens_workspace same path :293 will hit it once this PR lands. Library visualizers are already on inversion_with_data.
A apply_sparse_operator squares data.real/data.imag in the data's own dtype: complex64 data + complex128 noise gives data_term 700140000.0 vs exact 700140007.0; log_evidence +3.5; chunked path (sparse_terms_from_chunks, promotes to complex128) disagrees with one-shot. Reproduced — pre-existing precision path, exposed by the cache autoarray/dataset/interferometer/dataset.py:391-394 Low. Visibilities.from_fits and the float (N,2) constructor cast to complex128; only a caller passing a complex64 ndarray directly reaches it. complex128 data: all paths bit-equal.
C Replacing/mutating dataset.noise_map after apply_sparse_operator leaves noise_normalization (and data_term) stale: 7.35 vs 12.90 for σ 1→2. Reproduced — but pre-existing: W~, dirty image and the sparse curvature (4× off) were already stale in that scenario before phase 1; phase 1 adds two more stale scalars. autoarray/fit/fit_interferometer.py:142; noise_map is a plain attribute Low. No library or workspace path reassigns noise_map on a sparse dataset (all dataset.noise_map = sites are imaging, pre-operator).
D check_noise_map_real_imag_equal uses np.allclose default atol=1e-8, so σ_real=1e-9 vs σ_imag=2e-9 passes; sparse curvature then 1.5× dense, log_evidence −2.77. Reproduced — check predates phase 1 (1ae2f2f3, 2026-08-28); phase 1 moved it into the shared helper inversion_interferometer_util.py:58,61 Not reachable with Jy-unit data (sdp81 ALMA σ min 6e-5 → tolerance ≈1.7e-4 relative; factor-2 mismatch accepted only below ~1e-8).

Recommended follow-up (not done here — scope decision for the human):

  1. B (fix): in AbstractInversion.data_subtracted_dict raise a clear InversionException when self.data is None; add it (and TypeError) to the inversion_plots.py handlers; switch the two workspace scripts to fit.inversion_with_data. PyAutoArray corrective PR + two workspace PRs.
  2. D (cheap hardening, same PR as 1): atol=0.0 on both allclose/isclose calls.
  3. A (cheap, same PR): cast to complex128 before squaring in apply_sparse_operator, so one-shot == chunked for any input dtype.
  4. C: document as unsupported (mutating a dataset after apply_sparse_operator was already wrong) or make noise_map/data setters drop sparse_operator. Judgment call.

None of the four changes the parity or memory claims in this PR's body for complex128 data. Reproduction scripts: session scratchpad repro_{A,B,C,D}.py.

🤖 Generated with Claude Code

This was referenced Sep 30, 2026
Closed
Merged
Merged
@Jammy2211
Jammy2211 merged commit 092897e into main Sep 30, 2026
4 checks passed
@Jammy2211
Jammy2211 deleted the feature/interferometer-sparse-precomputed-data-term branch September 30, 2026 09:51
@Jammy2211 Jammy2211 mentioned this pull request Sep 30, 2026
Merged
5 of 6 tasks
@Jammy2211 Jammy2211 removed the pending-release PR queued for the next release build label Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat(interferometer): pass data=None on the sparse precomputed-data-term path (Discussion #13 lens parity)

1 participant

@Jammy2211